On the evaluation of Geographic Information Retrieval systems Evaluation framework and case study

Search engines for Digital Libraries allow users to retrieve documents according to their contents. They process documents without differentiating the manifold aspects of information. Spatial and temporal dimensions are particularly dismissed. These dimensions are, however, of great interest for users of search engines targeting either the Web or specialized Digital Libraries. Recent studies reported that nearly 20% queries convey spatial and temporal information in addition to topical information. These three dimensions were referred to as parts of ‘geographic information.’ In the literature, search engines handling those dimensions are called ‘Geographic Information Retrieval (GIR) systems.’ Although several initiatives for evaluating GIR systems were undertaken, none was concerned with evaluating these three dimensions altogether. In this article, we address this issue by designing an evaluation framework, whose usefulness is highlighted through a case study involving a test collection and a GIR system. This framework allowed the comparison of our GIR system to state-of-the-art topical approaches. We also performed experiments for measuring performance This paper is a substantially revised and extended version of a paper (with the title ‘Measuring Effectiveness of Geographic IR Systems in Digital Libraries: Evaluation Framework and Case Study’) originally appeared in the Proceedings of the 14th European Conference on Digital Libraries (ECDL 2010). Electronic Supplementary Material The online version of this article (doi:10.1007/s00799-011-0070-z) contains supplementary material, which is available to authorized users. D. Palacio · C. Sallaberry Université de Pau et des Pays de l’Adour, LIUPPA ÉA 3000 Avenue de l’Université, BP 1155, F-64013 Pau cedex E-mail: [email protected], [email protected] G. Cabanac · G. Hubert Université de Toulouse, IRIT UMR 5505 CNRS 118 route de Narbonne, F-31062 Toulouse cedex 9 E-mail: [email protected], [email protected] improvement stemming from each dimension or their combination. We show that combining the three dimensions yields improvement in effectiveness (+73.9%) over a common topical baseline. Moreover, rather than conveying redundancy, the three dimensions complement each other.